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TITLE OF THE INVENTION 
OBJECT REGION DATA GENERATING METHOD, OBJECT REGION 
DATA GENERATING APPARATUS, APPROXIMATION POLYGON 
GENERATING METHOD, AND APPROXIMATION POLYGON GENERATING 
5 APPARATUS 

CROSS-REFERENCE TO RELATED APPLICATIONS 
This application is based upon and claims the 
benefit of priority from the prior Japanese Patent 

n 

3 Application No. 2000-077161,^ filed March 17, 2000, the 

1% 10 entire contents of which are incorporated herein by 

i in; 

'5? reference. 

:J BACKGROUND OF THE INVENTION 

The present invention relates an object region 

r' ! 
" 15=" 

data generating method and apparatus for generating 
IB 15 data that indicates a region of an object in a moving 

I** picture. In addition, the present invention relates to 

an approximation polygon generating method and 
apparatus for generating an approximation polygon 
having an arbitrary object region approximated in one 
20 frame of a moving picture or a still picture. 

In recent years, with rapid advancement of image 
processing technology, a moving picture or a still 
picture is generally processed as digital data. With 
this digitization, a technique for efficiently 
25 compressing moving picture or still picture data with 

its large data quantity has been established. In 
addition, with the development of network technology 
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such as Internet or satellite broadcasting, CATV and 
the like, a moving picture or still picture database 
which stores a large amount of moving picture or still 
picture or a video-on-demand system will be put into a 
5 practical stage. Further, an automatic supervisory 

system from a remote site becomes a major object. In 
this way, when a moving picture or a still picture is 
used, there is a variety of demands to classify such 
^ moving picture or still picture according to what 

m 

^ 10 exists in a screen, and extract, process, and retrieve 

;^ an object in a moving picture or a still picture . 

32 In order to respond to these demands, technical 

]^ research for determining what is drawn in a moving 

^ picture or a still picture is made. However, both of a 

fp 15 background and a plurality of objects exist in a moving 

O 

!^ picture or a still picture. In order to identify a 

specific object, it is required to identify such 
background from objects as preprocessing. At this time, 
it is required to represent object region data in any 

20 shape, and data on object position, size, and shape 

must be precisely expressed. As a technique for this 
purpose, for example, there is developed an alpha map 
representing an image with high brightness of a portion 
at which an object exists and with low brightness of a 

25 portion at which no object exists or a bounding box 

approximating an object region in rectangular shape. 
However, alpha map has disadvantages that a large 



amount of data is required, and a large calculation 
amount of detection of a position or size of an object 
is required. In addition, in such bounding box, it is 
impossible to express a shape of an object. 

As a technique to solve these disadvantages, there 
is a method for approximating an object in a polygonal 
shape. In this case, only the coordinate of each 
vertex of a polygon is described, and thus, a 
remarkably smaller amount of data than that in alpha 
map is required. In addition, this method is the most 
suitable to expression of object region data because it 
has characteristics in which the position, size, and 
shape of an object is expressed faithfully. 

However, conventionally, as a method for 
approximating an object in a polygonal shape, there has 
been mainly used a technique in which an inertial main 
axis of an object is obtained, a point at which the 
inertial main axis cross an object contour is defined 
as a vertex, and then, a point on the object contour on 
which a distance from a line connecting vertexes to 
each other is the most distant is added as a vertex to 
obtain a polygon with a required number of vertexes. 
In this technique, it is required to search a point on 
an object contour every time an attempt is made to add 
one vertex. Thus, there has been a disadvantage that a 
large calculation amount of polygon approximation is 
required. In addition, a vertex always exists on an 



object contour, and thus, a segment of line connecting 
vertexes to each other enters the inside of an object 
at a protrusion portion of the object. Conversely, a 
segment of line connecting vertex to each other is 
expelled to the outside of an object at a recess 
portion of the object. Thus, there is a problem that 
large amount of error between a polygon and an object 
is required. 

In addition, with respect to expression of an 
object in a moving picture as well, polygon 
approximation is excellent as compared with alpha map 
or bounding box or the like from the aspect of data 
amount or expression of the position, size, and shape 
of the object for the reason similar to that in still 
picture. However, if a time-series variation of an 
object is expressed by polygon approximation, the 
movement of each vertex that is polygon approximated i 
described. However, in order to serially record a 
position of the vertex of each frame, there is a 
problem that a large amount of data is required, which 
is inconvenient. 

BRIEF SUMMARY OF THE INVENTION 

Accordingly, it is an object of the present 
invention to provide an object region data generating 
method and apparatus generating a trajectory of a 
vertex of an approximation polygon that expresses a 
region of a desired object in a moving picture with 
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high speed and high precision. 

It is another object of the present invention to 
provide an object region data generating method and 
apparatus describing a region of a desired object in a 
moving picture with a small amount of data, and 
facilitating generation of such object or data 
processing . 

It is still another object of the present 
invention to provide an approximation polygon 
generating method and apparatus generating an 
approximation polygon that approximates a region of an 
arbitrary object in one frame of a moving picture or a 
still picture with high precision and high speed. 

According to the present invention, there is 
provided a method of generating object region data 
relating to object regions in frames of moving picture, 
the frames lined up in a time-series variation, the 
method comprising : 

approximating the object region in each of the 
frames with a polygon, the polygon having vertexes; 

associating the vertexes in each of the frames 
with the respective vertexes in an adjacent frame of 
each of the frames; 

- obtaining trajectories linking the vertexes 
associated together based on the time-series variation 
of the frames; and 

generating the object region data based on the 
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trajectories . 

According to the present invention, there is 
provided another method of generating object region 
data relating to an object region in moving picture 
5 data having a plurality of frames, the method 

comprising : 

approximating the object region in each of the 
plurality of frames with a polygon; 

ig estimating a plurality of vertexes of the polygon 

fig 

10 in a given frame based on trajectory data indicating 

^2 vertexes of the polygon from a first frame to an 

;S immediately preceding frame of the given frame; 

t rw 

^ modifying the plurality of vertexes estimated so 

} * as to be associated with the object region in the given 

^ 15 frame; 

obtaining trajectory data indicating vertexes of 

the polygon from the first frame to the given frame; 

and 

generating the object region data based on the 
20 trajectory data indicating vertexes of the polygon from 

the first frame to the given frame. 

According to the present invention, there is 
provided a further method of generating a polygon that 
approximates an object region in an image, the method 
25 comprising: 

extracting a contour of the object region to 
generate an initial polygon; 



obtaining a characteristic quantity of each of 
vertexes of the initial polygon; 

selecting the vertex to be deleted based on the 
characteristic quantity; 

deleting the selected vertex; 

correcting a position of vertexes adjacent to the 
deleted vertex, thereby performing processing for 
updating the polygon; and 

repeatedly performing the processing until a 
predetermined ending condition has been met, thereby 
generating the polygon that approximates the object 
region . 

According to the present invention, there is 
provided an apparatus for generating object region data 
relating to object regions in frames of moving picture, 
the frames lined up in a time-series variation, the 
apparatus comprising : 

an approximation unit configured to approximate 
the object region in each of the frames with a polygon, 
the polygon having vertexes; 

an association unit configured to associate the 
vertexes in each of the frames with the respective 
vertexes in an adjacent frame of each of the frames; 

a trajectory unit configured to obtain 
trajectories linking the vertexes associated together 
based on the time-series variation of the frames; and 

a data generation unit configured to generate the 
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object region data based on the trajectories. 

According to the present invention, there is 
provided another apparatus for generating object region 
data relating to an object region in moving picture 
5 data having a plurality of frames, the apparatus 

comprising: 

an approximation unit configured to approximate 
the object region in each of the plurality of frames 
with a polygon; 

10 an estimation unit configured to estimate a 

plurality of vertexes of the polygon in a given frame 
based on trajectory data indicating vertexes of the 
polygon from a first frame to an immediately preceding 
frame of the given frame; 

15 a modifying unit configured to modify the 

plurality of vertexes estimated so as to be associated 
with the object region in the given frame; 

a trajectory unit configured to obtain trajectory 
data indicating vertexes of the polygon from the first 

20 frame to the given frame; and 

a data generation unit configured to generate the 
object region data based on the trajectory data 
indicating vertexes of the polygon from the first frame 
to the given frame. 

25 According to the present invention, there is 

provided a still another apparatus for generating a 
polygon that approximates an object region in an image, 



the apparatus comprising: 

an extraction unit configured to extract a contour 
of the object region to generate an initial polygon; 

a characteristic quantity unit configured to 
obtain a characteristic quantity of each of vertexes of 
the initial polygon; 

a selector configured to select the vertex to be 
deleted based on the characteristic quantity; 

a deletion unit configured to delete the selected 
vertex; 

a correction unit configured to correct a position 
of vertexes adjacent to the deleted vertex, thereby 
performing processing for updating the polygon; and 

a controller configured to repeatedly perform the 
processing until a predetermined ending condition has 
been met, thereby generating the polygon that 
approximates the object region. 

According to the present invention, there is 
provided an article of manufacture comprising a 
computer usable medium having computer readable program 
code means embodied therein, the computer readable 
program code means comprising: 

first computer readable program code means for 
causing a computer to approximate the object region in 
each of the frames with a polygon, the polygon having 
vertexes; 

second computer readable program code means for 
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causing a computer to associate the vertexes in each of 
the frames with the respective vertexes in an adjacent 
frame of each of the frames; 

third computer readable program code means for 
5 causing a computer to obtain trajectories linking the 

vertexes associated together based on the time-series 
variation of the frames; and 

fourth computer readable program code means for 
bp causing a computer to generate the object region data 

us. 

11 10 based on the trajectories. 

■j t== 

= g According to the present invention, there is 

provided another article of manufacture comprising a 
computer usable medium having computer readable program 
code means embodied therein, the computer readable 
15 program code means comprising: 

first computer readable program code means for 
causing a computer to approximate the object region in 
each of the plurality of frames with a polygon; 

second computer readable program code means for 
20 causing a computer to estimate a plurality of vertexes 

of the polygon in a given frame based on trajectory 
data indicating vertexes of the polygon from a first 
frame to an immediately preceding frame of the given 
frame; 

25 third computer readable program code means for 

causing a computer to modify the plurality of vertexes 
estimated so as to be associated with the object region 



in the given frame; 

fourth computer readable program code means for 
causing a computer to obtain trajectory data indicating 
vertexes of the polygon from the first frame to the 
given frame; and 

fifth computer readable program code means for 
causing a computer to generate the object region data 
based on the trajectory data indicating vertexes of the 
polygon from the first frame to the given frame. 

According to the present invention, there is 
provided a still another article of manufacture 
comprising a computer usable medium having computer 
readable program code means embodied therein, the 
computer readable program code means comprising: 

first computer readable program code means for 
causing a computer to extract a contour of the object 
region to generate an initial polygon; 

second computer readable program code means for 
causing a computer to obtain a characteristic quantity 
of each of vertexes of the initial polygon; 

third computer readable program code means for 
causing a computer to select the vertex to be deleted 
based on the characteristic quantity; 

fourth computer readable program code means for 
causing a computer to delete the selected vertex; 

fifth computer readable program code means for 
causing a computer to correct a position of vertexes 
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adjacent to the deleted vertex, thereby performing 
processing for updating the polygon; and 

sixth computer readable program code means for 
causing a computer to repeatedly perform the processing 
5 until a predetermined ending condition has been met, 

thereby generating the polygon that approximates the 
object region. 

According to the present invention, the size, 
tig shape, movement, and deformation of a region for a 

2 ¥3 

i*i 10 desired object in a moving picture is approximated by a 

polygon with high precision and high speed. In 
addition, a region for a desired object in a moving 
picture is described with a small amount of data, 
making it possible to facilitate such generation and 
15 data processing. As a result, in a moving picture 

database or a supervisory system and the like, the 
present invention is efficiently used for object 
recognition, classification or search and the like. 
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING 
20 FIG. 1 is a view showing an example of an object 

region data generating apparatus according to a first 
embodiment of the present invention; 

FIG. 2 is a view showing an example of processing 
procedures in the first embodiment; 
25 FIG. 3 is a flow chart showing an example of 

polygon generating processing; 

FIG. 4 is a view illustrating a polygon generating 
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processing; 

FIG. 5 is a view illustrating vertex deletion 
processing and moving processing of the adjacent 
vertexes in a processing for generating an 
5 approximation polygon; 

FIG. 6 is a flow chart showing an example of 
procedures for vertex association processing; 

FIG. 7 is a view illustrating a vertex association 

- car 

=p process; 

(3 10 FIG. 8A and FIG. 8B are views each illustrating 

,p function approximation processing; 

FIG. 9 is a view showing an example of a data 
± ~ structure of object region data; 

! ; 5 

:~ FIG. 10 is a view showing an example of a data 

]ts 15 structure of approximation polygon data contained in 

-■^ object region data; 

FIG. 11 is a view showing an example of a data 
structure of vertex trajectory data contained in 
approximation polygon data; 
20 FIG. 12 is a flow chart showing an example of 

processing procedures in a second embodiment of the 
present invention; and 

FIG. 13 is a view illustrating processing in the 
second embodiment of the present invention. 
25 DETAILED DESCRIPTION OF THE INVENTION 

A preferred embodiment of an object region data 
generating method and apparatus and an approximation 



polygon generating method and apparatus according to 
the present invention will now be described with 
reference to the accompanying drawings. 

FIG, 1 shows a configuration example of an object 
region data generating apparatus according to the first 
embodiment of the present invention. As shown in 
FIG. 1, this object region data generating apparatus 
comprises an image storage device 100 , a polygon 
generator 101, a vertex association unit 102, a 
function approximation unit 103, and an object region 
data storage device 104. 

The image storage device 100 stores image data 
such as moving picture data or still picture data, 
and comprises a hard disk or optical disk and a 
semiconductor memory or the like. 

The polygon generator 101 generates a polygon that 
represents a region of an object to be extracted from 
among moving picture data with respect to each frame at 
which such object exists. The term "object" used here 
denotes, for example, a specific person or animal in 
an image or any other creatures or manufactures or 
structural materials such as car or building (or parts 
of these materials including human head, car chassis, 
or gate of building) . An object is specified by an 
operator or an object extracting algorithm, such as 
used for a stereoscopic display. Hereinafter, a 
polygon representing an area for an object is called an 
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approximation polygon. An approximation polygon is 
represented by the coordinate (X-Y coordinate group) of 
each vertex of such polygon. 

There are two methods for generating an 
5 approximation polygon: a method generating polygons 

with their identical vertexes in number over all frames 
with regard to identical objects and a method in which 
the number of vertexes in a polygon may be different 
*g from another every frame. 

q 10 The vertex association unit 102 associates 

corresponding vertexes of approximation polygons 

m 

if- between adjacent frames in a linking manner with regard 

je= to the vertexes of approximation polygons generated 

over a plurality of frames with regard to each object, 

| E3 

15 and generates a trajectory of the vertexes of the 

approximation polygons over a plurality of frames 
with regard to each object. That is, this vertex 
association processing is performed for each object to 
be extracted from moving picture data from a frame in 

20 which such each object appears over a frame in which it 

disappears . 

For example, when a vertex of an approximation 
polygon in which a region of an object in a frame is 
approximated is associated with a vertex of the 
25 corresponding polygon of the adjacent frame, thereby 

obtaining a trajectory of the vertex, a distance 
between the vertex in one frame and the associated 



vertex of the adjacent frame may be determined so as to 
be minimum. Before obtaining this distance, two 
polygons may be aligned with each other so that their 
centers of gravity coincides with each other. 

In addition, for example, a position of a vertex 
of the adjacent frame is estimated from a trajectory of 
the already associated vertex so that a distance 
between the estimated vertex and a vertex in the 
adjacent frame may be determined so as to be minimum. 

Further, for example, the characteristic quantity 
in vertexes of a polygon is calculated so that a vertex 
having its closest characteristic quantity in the 
adjacent frame may be associated with another vertex. 

The function approximation unit 103 performs 
function approximation for each object with regard to 
the trajectory of each vertex in an approximation 
polygon (a series of the coordinates of the 
corresponding vertexes over a plurality of frames from 
a frame in which an object appears to a frame in which 
an object disappears) . For example, with respect to 
each vertex of an approximation polygon of one object, 
a trajectory of a vertex is expressed as a function of 
a frame number "f" or a time stamp "t" assigned to a 
moving picture for each of the X and Y coordinates. 
Function approximation' methods include approximation 
using a straight line or approximation using a spline 
curve and the like. The result of function 



approximation is expressed by a parameter value such as 
a polynomial coefficient of an approximation function, 
for example. In this manner, the trajectory of 
vertexes over a plurality of frames is approximated by 
a function and the function is expressed by only 
parameters, thereby a data quantity to be recorded is 
reduced. 

The object region data storage device 104 stores 
in accordance with a predetermined format, object 
region data generated by the function approximation 
unit 103, the data containing data obtained by 
expressing a time series trajectory of each coordinate 
of each vertex of the approximation polygon of each 
object in a function approximation manner (for example, 
parameter value such as polynomial coefficient of 
approximation function) . As with the image storage 
device 100 and the like, the object region data storage 
device 104 also comprises a hard disk or optical disk, 
a semiconductor memory and the like, for example. 

The object region data generated with respect to 
each object is used when an object is recognized or 
classified in a moving picture database or a 
supervisory system and the like. 

The image storage device 100 and the object region 
data storage device 104 may comprise different storage 
devices or identical storage devices. 

FIG. 2 is a flow chart showing an example of the 
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flow of processing using an object region data 
generating apparatus according to the present 
embodiment . 

A variety of processing procedures include: 
5 (1) a method of performing processing sequentially 

for each frame by reading one frame to perform a series 
of processing functions (polygon generation processing, 
vertex association processing, function approximation 
processing) at steps Sll to S13, and reading a next 
10 frame; 

(2) a method of first performing processing for 
performing polygon generation processing at step Sll 
with regard to a plurality of frames, performing vertex 
association processing at step S12 in batch, and 

15 performing function approximation processing at step 

S13; and 

(3) a method of first performing polygon 
generation processing at step Sll with regard to all 
the target frames, performing vertex association 

20 processing at step S12 in batch, and performing 

function approximation processing at step S13 in batch. 

The object region data generating apparatus 
according to the present embodiment can be implemented 
using software, of course. In this case, a program may 

25 be executed on a computer by creating the program in 

accordance with the flow chart shown in FIG. 2, for 
example, and installing it in the computer. 



Hereinafter, a description is mainly given to a 
case in which function approximation using the function 
approximation unit 103 is performed, thereby reducing a 
data quantity, and recording object region data. It is 
possible to record the coordinate value of each vertex 
of an approximation graphics for each frame in object 
region data without performing function approximation 
using the function approximation unit 103 (in the 
latter case, the function approximation unit 103 may 
not be provided) . 

Hereinafter, each of the processing functions in 
the present embodiment will be described in more detail. 

First, polygon generation processing using the 
polygon generator 101 will be described with reference 
to FIG. 3 to FIG. 5. 

FIG. 3 shows an example of processing procedures 
for generating an approximation polygon that indicates 
a region of an object from a frame in which an object 
exists . 

First, at step S21, an initial polygon precisely 
indicating a region of an object with a plurality of 
vertexes is generated. 

As input images, there may be used a bit map image 
(alpha map) in which a region in which an object exists 
is defined as "1" or a region in which an object does 
not exist is defined as "0"; an edge image obtained by 
removing a sharp edge from the image; or any other 



format image as long as a region in which an object 
exists or a region in which an object does not exist 
can be identified from each other. An example of input 
image is shown in an image 301 shown in FIG. 4. 

One point on an object contour is searched from 
the input image, and that point is defined as a vertex 
of an initial polygon. When this vertex is defined as 
a start point, while an object contour is traced with a 
short distance, the vertex on the contour is defined as 
a vertex, and is sequentially added to the initial 
polygon. In general, although the initial polygon is 
formed as a polygon with a very large number of 
vertexes, a polygon approximation error (for example, 
difference between an area for an actual object region 
and an area for the approximation polygon or a value 
obtained by dividing this difference by the area for 
the actual object region or the area for the 
approximation polygon) is small. An example of the 
initial polygon obtained by the input image 301 is 
shown in an image 302 shown in FIG. 4. In this case, a 
large amount of data is required, and then, processing 
for reducing such amount of data is performed. 

At step S22, a characteristic quantity is 
calculated with regard to each vertex of the obtained 
initial polygon. The characteristic quantity estimates 
how much polygon approximation error increases when 
such vertex is deleted. The characteristic quantity 



may be an angle formed at such vertex and two vertexes 
adjacent thereto or an area for a triangle formed at 
such vertex and two vertexes adjacent thereto. 

An example of a relationship between a vertex and 
its characteristic quantity is shown in an image 304 
shown in FIG. 5. In the image 304, an angle formed at 
a vertex t^ and two vertex adjacent thereto is defined 
as Q ±r anc * an area for a triangle formed at the vertex 
tj_ and two vertex adjacent thereto is defined as s-^. 

Where an area s± for a triangle is used as a 
characteristic quantity of vertex tj_ = (x-^, y-jj , when 
the adjacent vertexes are defined as tj__]_ = (x^-^, 
yj_-l) , + = (x-j_+]_, Yi + l) f the characteristic quantity 
Si is represented by: 

si = | (x i+1 - x ± ) • (yi-i, y±) - (xi-x, x ± ) • 
(Yi+l. Yi) I / 2 

If a polygon before deleting a vertex t^ well 
approximates an object, an approximation error 
increased by deleting the vertex t^ is proportional to 
the characteristic quantity Sj_. Therefore, if the 
characteristic quantity s-^ is small, even if the vertex 
t± is deleted, the approximation error is less affected. 

At step S23, one vertex at which the smallest 
polygon approximation error occurs when the vertex is 
deleted (that is, a vertex having its minimum 
characteristic quantity s-jj is selected from the 
characteristic quantity obtained at step S22, and such 
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vertex is deleted. 

If a deleted point corresponds to a protrusion 
portion of a polygon, there is high possibility that 
the approximation error produced by deleting a vertex 
corresponds to an error produced by a polygon entering 
the inside of an object. In this case, the two 
vertexes adjacent to the deleted vertex are moved to 
the outside of the polygon, thereby reducing an 
approximation error to the minimum. 

Conversely, if a deleted point corresponds to a 
recess portion of a polygon, there is high possibility 
that the approximation error produced by deleting a 
vertex corresponds to an error produced by the polygon 
being expelled to the outside of the object. In this 
case, therefore, two vertexes adjacent to the deleted 
vertex is moved to the inside of the polygon, thereby 
reducing an approximation error to the minimum. 

Although a movement vector of the adjacent 
vertexes may be arbitrary, if the movement quantity is 
too large, the approximation error increases conversely. 
Because of this, for example, this vector may be 
determined from a polygon approximation error or its 
peripheral vertex coordinates. 

In the case of determination from the polygon 
approximation error, while a movement vector is changed, 
the best movement vector is searched for (for example, 
a movement vector with its minimum error is adopted) . 



In the case of determination from its peripheral 
vertex coordinate, for example, when a vertex t^ is 
deleted, a movement vector v^ of an adjacent vertexes 
ti-1 and t^+i ^ s determined as a vector vertical to a 
straight line connecting a vertex tj_ + i to a vertex tj__i 
the vector having a length that is half of a distance 
between the straight line connecting the vertex t^-f^ to 
the vertex tj__]_ and a vertex t^. When a vector 
connecting the vertex t^ to the vertex tj__]_ is defined 
as aj_, and a vector connecting the vertex t-j_+]_ to the 
vertex t-^-i is defined as b^, the movement vector vj_ is 
expressed as follows. 

v ± = { a ± (ai-bi / |b ± |2) x b ± } / 2 
where (aj_ • b^ / |bjj2) x bj[ denotes a positive 
projection of a^ and b^ . 

An example of a movement vector of the adjacent 
vertex when a vertex is deleted is shown in an image 
305 shown in FIG. 5, and an example when the adjacent 
vertexes are moved by movement vector is shown in an 
image 306 shown in FIG. 5. 

The vertexes are deleted, and the adjacent 
vertexes are moved, whereby there is a need to update 
the characteristic quantity of a vertex. At step S24, 
the characteristic quantity of a vertex to be updated 
is re-calculated. The vertex to be updated is a vertex 
moved at step S23 (vertexes tj__i and t^ + i shown an 
example shown in FIG. 5) and the adjacent vertexes 



thereto an <3 ti+2 ^ n an example shown in FIG. 5) . 

At step S25, it is determined whether or not 
termination conditions are met. 

A variety of termination conditions will be 
described below. 

For example, there is a method of using a 
termination condition as to whether the number of 
vertexes in an approximation polygon is many more than 
a specified number. In this case, if the polygon 
includes the specified number of vertexes or less as 
a result of deletion of vertexes, processing is 
terminated. If vertexes equal to or greater than the 
specified number of vertexes still remains, it returns 
to step S23 at which steps S23 to S25 are repeated 
until the polygon includes the specified number of 
vertexes or less. 

In addition, apart from a method of judgment 
according to whether or not a polygon includes a 
specified number of vertexes or less, for example, 
there is a method for specifying a termination 
condition according to whether or not a polygon 
approximation error exceeds a threshold. In this case 
if the polygon approximation error exceeds such 
threshold, processing terminates. 

Further, an AND or OR combination or the like of 
plurality of conditions may be employed. 

Where function approximation is performed using 
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the function approximation unit 103, it is preferable 
that the number of vertexes be equal (there are a 
method in which the number of vertexes of all objects 
are identical to each other and a method in which the 
5 number of vertexes of objects may be different from 

each other) . Conversely, if function approximation is 
not performed using the function approximation unit 103, 
the number of vertexes. may not be equal. 

An example of an approximation polygon that 

10 represents a region of an object obtained by this 

processing is shown in an image 303 shown in FIG. 4. 

Now, vertex association processing in the adjacent 
frames using the vertex association unit 102 will be 
described with reference to FIG. 6 and FIG. 7. 

15 FIG. 6 shows an example of processing procedures 

for associating each vertex of an approximation polygon 
that represents a region for each object extracted for 
each frame . 

First, at step S71, when an approximation polygon 
20 representing a region of an object in a frame is 

defined as P^, and an approximation polygon in a next 
frame (adjacent frame) of such frame is defined as Pi+i, 
the centers of gravity of these polygons P-^ and Pi+i 
are obtained. Examples of the approximation polygons 
25 Pj_ and Pi + i, and the center of gravity obtained 

therefor are shown in the images 311 and 321 in FIG. 7. 
If there are plural objects (its approximation 
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polygons) in the adjacent frame, a distance between the 
center of gravity of the approximation polygon P-^ of 
the object targeted in the frame and that of each of 
the approximation polygons in the adjacent frame is 
5 calculated, and then, an approximation polygon having 

its center of. gravity that is the shortest in distance 
among them is defined as an approximation polygon Pi+i 
that corresponds to the object. Of course, the 
h Q approximation polygons may be associated with each 

m 10 other using one variation of another method. 

m \ 

Next, consider a polygon P' -j_ in which the center 

\ 

1 I of gravity of ^:he approximation polygon Pj_ in the frame 

! J ^ v \ 

]^ is moved to be positioned at the center of gravity of 

\- the approximation^ polygon Pi+i in the adjacent frames, 

llj. 15 An example of a relationship between each of the 

H polygons Pj_, Pi+i, aod P' ^ and each of their centers of 

gravity is shown in in\ages 312 and 322 shown in FIG. 7. 

However, if a distance between the center of 
gravity of the approximation polygon P-^ and that of the 
20 approximation polygon Pi+i is short, the center of 

gravity of the approximation polygon P-j_ may be defined 
as P' i without causing movement such that the center of 
gravity of the approximation polygon Pj_ coincides with 
that of the approximation polygon Pi+i- 
25 At step S73, each vertex of the polygon P' j_ is 

associated with each vertex of Pi+i . An example in 
which these vertexes are associated with each other is 



shown in images 313 and 323 shown in FIG. 7. 

Hereinafter, examples of a variety of vertex 
association processing functions will be shown below* 

(Method 1: Method for determining based on 
distance between vertexes) 

A distance between a vertex and each vertex of the 
approximation polygon Pi+i in the adjacent frame is 
obtained for each polygon P' ^ which has been moved to 
a position at which the center of gravity thereof 
coincides with that of the approximation polygon Pj_+i, 
and a vertex of the polygon Pi+i with its shortest 
distance is defined as a vertex corresponding to such 
vertex . 

As a result of the above processing, if one or 
more of the vertexes of the polygon Pi+i cannot be 
associated with vertexes of the polygon P' j_ among the 
vertexes of the polygon Pi + i, such vertex is associated 
with a vertex that is the closest to the vertexes of 
the polygon P' j_ (this applies to methods 2 and 3). 

(Method 2: Method for determining association 
based on the shape of a polygon) 

The characteristic quantity of each vertex of an 
approximation polygon in each frame is obtained, and 
the vertexes with their close characteristic quantities 
are associated with each other. The characteristic 
quantities include an area or a vertex angle of a 
triangle produced at the vertex and the adjacent two 
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vertexes thereto. 

(Method 3: Method for estimating a vertex position 
of a frame to be associated based on the trajectory of 
vertex) 

5 The trajectory of j-th vertex of the vertexes of 

polygons P]_, . . . , P-^ associated in frames is 
approximated by a time function Tj (t) ; a vertex 
position in the frame i + 1 is estimated based on the 
function Tj (t) ; and a vertex that is the closest to the 
10 estimated vertex position, of the vertexes of the 

polygon Pi+i in the frame, is defined as a vertex that 
corresponds to the j-th vertex. 
(Method 4) 

Where the number of vertexes in the polygon P' j_ is 
15 equal to that of the polygon Pi+i, in the above methods 

(1 to 3) if plural vertexes of the polygon P'j_ are 
associated with same vertex in the polygon Pi+i as a 
result of the processing (that is, if one or more 
vertexes in the polygon Pi+i cannot be associated with 
20 the vertexes of the polygon P'i), such association may 

not be used. In addition, processing may be continued 
in order to find out another association such that the 
vertexes of the polygon P' ± are associated with those 
of the polygon Pi+i one-to-one. For example, if plural 
25 vertexes of the polygon P' j_ are to be associated with 

the same vertex of the polygon Pi+i, one proper vertex 
of the plural vertexes of the polygon P'i is excluded 
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(to be associated with the vertex of the polygon P-l+i), 
and the remaining vertexes are tried to be associated 
with one that is not associated with the vertex of the 
polygon P' j_ of the vertexes of the polygon Pi+i- 
5 (Method 5) 

In the above methods (1 to 4), in the case of 
limitation to association such that there does not 
occur a portion at which the sequence of vertex 

I** 

!q association is replaced with vertex association (for 

10 example, a case in which the vertexes of the polygons 

^ in the mutually adjacent frames correspond to each 

if; 

:^ other in the clockwise sequence) , when association is 

obtained such that the sequence of vertex association 

E £ 

5^ is not reversed, such association is not be used. Then, 

10 15 processing is continuous so as to find out another 

O 

!<£ association in the sequential order. For example, a 

part or whole of the association is changed, processing 
is repeated in a trial and error manner. 
(Method 6) 

20 Where the number of vertexes in a polygon P' j_ is 

equal to that of a polygon Pi+]_, if the sequence of 
vertexes is limited to association in a predetermined 
order such as the clockwise sequence, for example, 
there is obtained a shortest distance between each of 

25 the vertexes of the polygon P'j_ and each of those of 

the polygon Pj_ + i, the vertex of the polygon P' ^ is 
associated with the. vertex of the polygon P'-^ + i, and 



further, the right adjacent vertex of the associated 
vertex of the polygon P' j_ is associated with the right 
adjacent vertex of the associated vertex of the polygon 
p i+l' and subsequently, such association may be 
repeated similarly in the clockwise direction. In 
addition, instead of obtaining one vertex first 
associated, there is obtained a sum of a distance 
between vertexes in the case of associating a 
predetermined number (a plurality) of vertexes adjacent 
to each other in the polygon P' with the same number 
of vertexes adjacent to each other in the polygon Pi + i, 
a correspondence of the predetermined number of 
vertexes in which such sum is minimum is adopted. 
Further, the right adjacent vertex of the associated 
predetermined number of vertexes of the polygon P' j_ is 
associated with that of the polygon Pj_ + i, and 
subsequently, association may be repeated similarly in 
the clockwise direction. Apart from use of a distance 
between vertexes for determining a first vertex 
association, as described above, there can be used a 
method for using a characteristic quantity or a method 
for using an estimated vertex position. 
(Method 7) 

In addition, if the number of vertexes in the 
polygon P' j_ is equal to that in the polygon Pi + i, in 
one variation of the above method 1, there is obtained 
a distance between each of the vertexes of the polygon 



P' i which is obtained by moving the approximation 
polygon in the frame and each of the vertexes of the 
approximation polygon Pi+i in the adjacent frame, so 
that there may be adopted a combination of association 
of vertexes such that a sum of distances between all of 
the vertexes is minimized, of combinations in which the 
vertexes of the polygon P' j_ are associated with those 
of the polygon Pi+i by one-to-one. With respect to the 
above methods 2 and 3, a similar variation is 
applicable. In method 2, the sum of a difference in 
characteristic quantity may be employed, and in method 
3, the sum of a distance between an estimated vertex 
position and an association vertex position may be 
employed. 

Further,, at this time, it is possible to consider 
a condition in which any sequence is not changed as 
described above, a condition based on one-to-one 
correspondence, or a condition based on one-to-one 
correspondence without changing the sequence of 
vertexes . 

With respect to one object, if the number of 
vertexes in an approximation polygon that represents a 
region for the object may differ depending on a frame 
in a duration from a frame in which an object appears 
to a frame in which an object disappears, the following 
processing is performed so that the above case can be 
processed in a manner similar to a case in which the 
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number of vertexes in the polygon P' ^ is equal to that 
in the polygon Pi+i- 

It is assumed that the number of vertexes for 
obtaining a trajectory, i.e., the number of vertexes of 
an approximation polygon that represents a region for 
an object is defined as "s" , and the number of vertexes 
in the approximation polygon in each frame is equal to 
or smaller than "s". When a frame has a number of 
vertexes s - As (As > 0), one of the vertexes of the 
approximation polygon in that frame is regarded as a 
multiplexed vertex of the As + 1 number of vertexes 
with their identical coordinates (alternatively, As 
vertexes are regarded as a multiplexed vertex of two 
vertexes with their identical coordinates or the other 
association using a similar technique is performed) , 
whereby processing (methods 1 to 7) can be used in a 
manner similar to a case in which the number of 
vertexes is equal. There are a method for performing 
processing after determining vertexes to be processed 
in advance as being multiplexed, and a method in which 
a vertex processed as being multiplied is determined as 
a result of processing. 

It is assumed that the number of vertexes for 
obtaining a trajectory, i.e., the number of vertexes of 
an approximation polygon that represents a region for 
an object is defined as "s", and the number of vertexes 
in the approximation polygon in each frame is larger 
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than "s" . When a frame has a number of vertexes s + 
As (As > 0), the coordinates with regard to 
unassociated As number of vertexes of the 
approximation polygon of that frame is specially 
recorded as an isolate point; or such data is aborted 
as an isolate point (the vertexes are processed as an 
absence) , whereby processing (methods 1 to 7) can be 
used in a manner similar to a case in which the number 
of vertexes is equal (there is a method for performing 
processing after determining vertexes to be regarded in 
advance as an isolate point, and a method in which a 
vertex regarded as an isolate point are determined as a 
result of processing) . 

In addition, if the number of vertexes for 
obtaining a trajectory, i.e., the number of vertexes of 
an approximation polygon that represents a region for 
an object is defined as "s", and the number of vertexes 
in the approximation polygon in each frame changes from 
a value smaller than "s" to a value larger than "s" , 
the above processing may be selectively performed in 
accordance with whether or not the number of vertexes 
of the approximation polygon in the frame is equal to 
or smaller than "s" or exceeds "s" . 

Now, function approximation processing using a 
function approximation unit 103 will be described below. 

When the above polygon approximation and vertex 
association completes with regard to all frames in 
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which objects exist, the trajectory of each coordinate 
of each vertex in the approximation polygon of each 
object is approximated by using a function, thereby 
reducing an amount of data. As function approximation, 
5 there may be employed any approximation technique such 

as linear approximation, quadric approximation, or 
spline function approximation. 

FIG. 8A and FIG. 8B are views each illustrating in 
more detail an outline of processing for approximating 
10 the trajectory of a vertex in an approximation polygon 

using the function approximation unit 103. 

In FIG. 8A, reference numeral 331 designates one 
;^ frame in a moving picture targeted for processing, 

j** Reference numeral 332 denotes each vertex when a region 

ijl 15 for an object is approximated by a polygon. Here, 

approximation is assumed to be done by a rectangle 
(four vertexes) in a series of frames for clarify. In 
addition, a description will be given, assuming that a 
spline function is used for function approximation. 
20 In function approximation processing, a trajectory 

obtained when each vertex of an approximation polygon 
that represents each object is arranged in time series 
is approximated based on a spline function for each of 
the X and Y coordinates. In this example, the spline 
25 function in the X and Y coordinates is required for 

each of the four vertexes that are vertexes of an 
approximation polygon. Thus, a total of eight spline 
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functions are generated. 

Processing using the function approximation unit 
103 include: a method for performing processing every 
time the coordinate value of the vertex of the 
approximation polygon in each frame concerning a region 
for the object is obtained (for example, a method for 
performing approximation every time the coordinate 
value of the vertex in each frame is obtained, and 
obtaining an approximation error, and then, properly 
setting an approximation interval, i.e., a frame 
interval so that such approximation error is lowered 
within a predetermined range) ; or a method for 
performing processing after the coordinate values of 
vertexes with regard to all frames concerning a region 
for the object are obtained. 

In addition, depending on a method for performing 
vertex association processing using the vertex 
association unit 102, there are a case in which a 
trajectory of vertexes over all frames from a frame in 
which a region for the object appears to a frame in 
which it disappears is obtained (for example, a case in 
which the number of vertexes of the association polygon 
is equal over all the frames, and a vertex is 
associated with another vertex by one-to-one among all 
adjacent frames) ; and a case in which a trajectory of a 
vertex existing only at an interval in some of all the 
frames from a frame in which a region for the object 
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appears to a frame in which it disappears (for example, 
a case in which the number of vertexes in an 
approximation polygon is different depending on the 
frames, and the trajectory of certain vertexes in a 
5 frame with a small number of vertexes is interrupted; 

or a case in which the trajectory of certain vertexes 
in such frame is interrupted because the corresponding 
vertexes in a frame correspond to each other by one-to- 
one, even if the number of vertexes in an approximation 

10 is equal over the all frames) . In the latter case, the 

frame in which the trajectory of vertexes is 
interrupted is always defined as a node of an 
approximation function . 

In FIG. 8B, reference numeral 333 represents an 

15 approximated spline function in a three-dimensional 

manner . 

Data indicating the thus obtained approximation 
function is recorded in the object region data storage 
device 104 in accordance with a predetermined data 

2 0 format. 

Hereinafter, the data format of object region data 
stored in the object region data storage device 104 
will be described. A description will be given by 
exemplifying a case in which a vertex is approximated 

25 by a spline function. Of course, this is applied to a 

case in which such vertex is approximated by another 
function . 
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FIG. 9 shows an example of the data format of 
object region data for recording a spline function that 
represents a region of an object in a moving picture. 

ID number 400 is an identification number assigned 
5 to each object. 

A starting frame number 401 and an ending frame 
number 402 are first and last frame numbers that define 
the existence of an object given by the ID number. 
Specifically, these numbers correspond to the numbers 

10 of frames in which an object appears and disappear in a 

moving picture. However, instead of the numbers of 
frames in which an object appears and disappears in a 
moving picture in reality, for example, an arbitrary 
frame number later than that assigned when an object 

15 appears in a moving picture may be assigned as a 

starting frame number. Similarly, an arbitrary frame 
number later than the starting frame number and earlier 
than that assigned when an object disappear in a moving 
picture may be assigned as an ending frame number. The 

20 starting/ending frame numbers can be substituted by the 

starting/ending time stamps. 

Approximation polygon data 403 is trajectory data 
on vertexes for expressing an approximation polygon 
(for example, parameter for spline function or the 

25 like) . Where one object is formed in a plurality of 

object regions, the approximation polygon data 403 is 
provided in plurality. 
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FIG. 10 shows an example of a data structure of 
approximation polygon data 4 03. 

The number of vertexes 431 indicates the number of 
vertexes for expressing an approximation polygon. 
5 A pair of vertex trajectory data 432-1 and 432-2 

are data regions concerning a spline function for 
expressing a trajectory of points representative of an 
approximation polygon. The spline function data on the 
^ X and Y coordinates is required for one vertex. 

™ 10 Therefore, the vertex trajectory data for specifying 

^0 such spline function exists by vertex number MX 2 . 

=S The sequence of vertexes to be recorded may be a 

e predetermined sequence such as a sequential order of 

fsj the vertexes, for example. 

ITT 15 FIG. 11 shows an example of a data structure of 

iT vertex trajectory data 432-1 and 432-2. 

A node frame number 4320 represents a node of a 
spline function. This number indicates that polynomial 
data 4323 is effective up to this node. The number of 
20 polynomial coefficient data varies depending on the 

maximum order of the spline function. If the maximum 
order is defined as K, the number of coefficient data 
is defined as K + 1 . Thus, reference is made to a 
polynomial order 4321. The polynomial order 4321 is 
25 followed by polynomial coefficient data 4322 in number 

that corresponds to K + 1. 

In addition, a spline function is expressed by a 



i 



m 



- 39 - 

polynomial between each of adjacent nodes, thus 
requiring polynomials in number that corresponds to the 
number of nodes. Therefore, data 4323 containing a 
node frame number 4320, polynomial coefficient 4322 or 
5 the like is repeatedly described in plurality. Where a 

node frame number is equal to a last frame, it means 
that the data 4323 is the last polynomial coefficient 
data. Thus, it is found that trajectory data ends. 
As in the above mentioned variations, if the 

if: 

!i 10 coordinate is recorded without causing association with 

i 

■g regard to a vertex of an approximation polygon in an 

:Jf adjacent frame, the coordinate of such vertex and its 

frame number are described in data 4323. 
IftJ. In the meantime, a variety of other items of data 

IP 15 may be added to the object region data shown in FIG. 9. 

For example, there may be added a variety of attribute 
data associated with an object of the ID number or 
data (such as file name, URL or any other address) 
indicating a method of access to associated information 
20 data associated with an object of the ID number. 

Associated information data may be characters, voices, 
still pictures, moving pictures or a combination 
thereof; or data describing programs or computer 
activities . 

25 Where function approximation is not performed 

using a function approximation unit 103, for example, 
the coordinate value of each vertex of an approximation 
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graphics in each frame may be recorded as it is in 
approximation polygon data 403 contained in the object 
region data shown in FIG. 9. 

Now, a method for providing moving picture data or 
object region data will be described below. 

Where the object region data created by the 
processing according to the present embodiment is 
provided for a user, it is required to provide object 
region data by using any method from a creator to a 
user. The data is provided in a variety of modes as 
exemplified below. 

(1) moving picture data and its object region data 
are provided simultaneously by being recorded in one or 
plural recording media; and 

(2) moving picture data is provided first, and 
then, object region data is provided simultaneously by 
being recorded in one or plural recording media. 

The above modes are associated with a case in 
which the data are mainly provided by a recording 
20 medium. Otherwise, there can be provided a mode in 

which part of whole of the data is provided via a 
communication medium. 

As has been described above, according to the 
present embodiment, there is provided an object region 
25 data generating method and apparatus for generating 

object region data that describes data concerning a 
region for an arbitrary object in moving picture data, 
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wherein the region for the object in the moving picture 
data is approximated by using an approximation polygon 
with regard to each of the plural frames, each of 
plural vertexes of the approximation polygon in a 
5 preceding frame is approximated with a vertex of the 

approximation polygon in the current frame having a 
shortest distance between vertexes after the 
approximation polygon in the current frame is move to 
make the centers of gravity of both approximation 

10 polygons coincide with each other, whereby a trajectory 

of the associated vertexes is obtained by arranging the 
vertexes along the sequence of the frames. The object 
region data is generated based on the trajectory of the 
obtained plural vertexes. 

15 According to the present embodiment, there is also 

provided an object region data generating method and 
apparatus for generating object region data that 
describes data concerning a region for an arbitrary 
object in moving picture data, wherein the region for 

20 the object in the moving picture data is approximated 

by using an approximation polygon with regard to each 
of the plural frames, each of plural vertexes of the 
approximation polygon in a preceding frame is 
approximated with a vertex of the approximation polygon 

25 in the current frame which is closest to an estimated 

position of the vertex based on a trajectory of the 
associated vertexes, whereby a trajectory of the 



associated vertexes is obtained by arranging the 
vertexes along the sequence of the frames. The object 
region data is generated based on the trajectory of the 
obtained plural vertexes. 

According to the present embodiment, there is 
provided an object region data generating method and 
apparatus for generating object region data that 
describes data concerning a region for an arbitrary 
object in moving picture data, wherein the region for 
the object in the moving picture data is approximated 
by using an approximation polygon with regard to each 
of the plural frames, each of plural vertexes of the 
approximation polygon in a preceding frame is 
approximated with a vertex of the approximation polygon 
in the current frame having a closest characteristic 
quantity, whereby a trajectory of the associated 
vertexes is obtained by arranging the vertexes along 
the sequence of the frames. The object region data is 
generated based on the trajectory of the obtained 
plural vertexes. 

A vertex association may be limited to a one-to- 
one association or such an association that the 
sequence of vertexes is not changed. 

According to the present embodiment, there is 
provided an approximation polygon generating method and 
apparatus for generating an approximation polygon that 
approximates a region of an arbitrary object in one 
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frame of a moving picture or an image, wherein a 
contour of a region of the object in one frame of a 
moving picture or a still picture is extracted; an 
initial polygon representing a region of the object is 
5 generated based on the extracted contour; a vertex to 

be deleted is selected based on the characteristic 
quantity of each vertex of a polygon at that time; the 
thus selected vertex is deleted; vertexes adjacent to 
the deleted vertex are moved; and the characteristic 

10 quantity of each vertex of the polygon is updated, and 

the processing is performed until a predetermined 
ending condition has been met, thereby generating an 
approximation polygon that approximates the region for 
the object in the frame. 

15 As the characteristic quantity, there may be used 

an area for a triangle produced by the deleted vertex 
and the adjacent two vertexes thereto or an angle 
produced by the above deleted vertex and the adjacent 
two vertexes thereto. 

20 As the ending condition, there may be used whether 

or not the number of vertexes in a polygon becomes 
smaller than a specified value or whether or not the 
approximation error value of the approximation polygon 
and the actual object is greater than the value. 

25 According to the present embodiment, an object in 

a moving picture is represented as a trajectory of 
vertexes in an approximation polygon as follows. 



A region in which an object exists in each frame 
is first detected, and an initial polygon is generated 
from a contour of such region. A vertex less affected 
by, deletion is searched for in a polygon based on the 
characteristic quantity of each vertex, and is deleted. 
At this time, an increase of an error caused by vertex 
deletion is reduced by moving the adjacent vertexes. 
This processing is repeated until the ending condition 
has been met, thereby producing a polygon that 
represents object region data with a smaller number of 
vertexes . 

Since a point on an object contour is searched for 
with a large amount of calculation only when an initial 
polygon is produced, this algorithm operates at a 
higher speed than a conventional algorithm. In 
addition, there is an advantage that an approximation 
error is small because a polygon is produced while an 
vertex is dynamically changed. 

Region data with regard to each frame is generated, 
and vertexes are associated with each other between the 
adjacent frames, whereby a time-series variation of 
object is described as a trajectory of vertexes. For 
example, association is performed so that a distance 
between the corresponding vertexes is minimized. Then, 
the trajectory of each vertex is approximated by 
function, whereby an amount of data is reduced. 

In addition, if an attempt is made to generate a 



trajectory of a vertex every time the object of each 
frame is inputted, the object region data on a frame to 
be generated is estimated based on the object region 
data on the already created frame, and the estimated 
object region data is deformed according to an object, 
whereby the trajectory of a vertex is produced. 

In this manner, according to the present 
embodiment, it is possible to approximate the size, 
shape, movement, deformation or the like, of a region 
of a desired object in a moving picture, as a polygon, 
with high precision and high speed. In addition, the 
region of the desired object in the moving picture is 
described in a small amount of data, making it possible 
to facilitate generation of such object and data 
processing. As a result, in a moving picture database 
or a supervisory system and the like, this approximated 
polygon is used for object recognition, classification, 
search or the like. 

In addition, according to the present invention, 
preferable function approximation is performed even if 
there is a possibility that the number of vertexes is 
different depending on each frame or there is a high 
possibility that vertexes between frames are associated 
with each other by object deformation or movement. 

As has been described above, it is possible to 
approximate the size, shape, movement, deformation or 
the like, of a region of a desired object in a moving 
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picture, as a polygon, with high precision and high 
speed. In addition, the region of the desired object 
in the moving picture is described in a small amount of 
data, making it possible to object generation or data 
5 processing. As a result, in a moving picture database 

or a supervisory system and the like, this approximated 
polygon is used for object recognition, classification, 
search or the like. 

^ In addition, according to the present embodiment, 

O- 

2 10 preferable function approximation is performed even if 

B there is a possibility that the number of vertexes is 

B different depending on each frame or there is a plural 

possibilities that the vertexes between frames are 
y associated with each other due to an object deformation 

fi 15 or movement. 

T Further, the region for the object in the moving 

picture is described as a parameter for a curve that 
approximates a time series trajectory (trajectory of 
the coordinate of vertexes at which a frame number or a 

20 time stamp is defined as a variable) . In this manner, 

the region of the object in the moving picture is 
expressed by a function parameter, and thus, there can 
be produced object region data in a small amount of 
data, the data being easily processed. In addition, an 

25 approximation polygon is easily generated from the 

parameter for the approximation parameter. 

The second embodiment of the object region data 
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generating method and apparatus and the approximation 
polygon generating method and apparatus according to 
the present invention will now be described with 
reference to the accompanying drawings. 

The present embodiment is basically similar to the 
first embodiment. Hereinafter, differences from the 
first embodiment will be mainly described. 

In the first embodiment, vertexes of an 
approximation polygon is obtained from a region of the 



52 10 object in each frame, and then, a correspondence of the 



vertexes of the approximation polygon is obtained in 
the adjacent two frames. In the second embodiment, 
each of the corresponding vertexes of an approximation 
polygon in a targeted frame is obtained by using the 

15 trajectory of each of the corresponding vertexes of the 

approximation polygon in the already obtained preceding 
frames. For example, this embodiment is efficient in a 
case in which processing is performed while a moving 
picture is inputted in a stream moving picture. 

20 A configuration example of an object region data 

generating apparatus according to the present 
embodiment is similar to that shown in FIG. 1. 

In addition, an example of processing flow using 
the object region data generating apparatus according 

25 to the present embodiment is similar to that shown in 

FIG. 2. However, the polygon generation processing at 
step Sll is performed only for the first frame in which 



an object appears. Alternatively, with respect to a 
first predetermined number of frames (several frames), 
a correspondence of a vertex of an approximation 
polygon of the current frame and that over a plurality 
of frames may be obtained in accordance with the 
processing at steps Sll and S12 in the first embodiment 
Subsequently, the vertexes of the approximation polygon 
in the subsequent frames are obtained one after another 
in accordance with the processing at step S12 in the 
present embodiment . 

Hereinafter, vertex association processing using a 
vertex association unit 102 in the present embodiment 
will be described with reference to FIG. 12 and FIG. 13 

At step S81, with respect to an object, the 
trajectory of each vertex of an approximation polygons 
in a plurality of already generated frames (for example 
all frames or a predetermined number of closest frames) 
is approximated as a time-series function, as 
designated by reference numeral 601 shown in FIG. 13. 
This function approximation may be performed by 
using any approximation technique such as linear 
approximation, quadric approximation or spline 
approximation . 

Next, at step S82, the vertex position in a frame 
to be processed is estimated based on an approximated 
function, thereby generating a vertex (vertexes) of a 
polygon estimated in the frame to be processed. An 




49 - 



example of the estimated approximation polygon is 
designated by reference numeral 602 shown in FIG. 13. 

If an object in each frame moves in a small 
quantity, a vertex (vertexes) of an approximation 
5 polygon in the closest frame is (are) employed intact 

as a vertex (vertexes) of an estimated polygon without 
performing function approximation. 

In addition, if an object in each frame moves in 
large quantity, but deformation is small in quantity, 

10 the center of gravity of the approximation polygon in a 

frame to be generated is estimated from that of each 
of the approximation polygons in a plurality of the 
already generated frames, and then, a polygon obtained 
by moving an approximation polygon in the closest frame 

15 so that its center of gravity coincides with the 

estimated center of gravity may be used as an estimated 
approximation polygon . 

At an initial stage of processing, if a small 
number of frames have been processed, the processing at 

20 steps S81 and S82 may be performed by a small number of 

such frames. Alternatively, an approximation polygon 
in the immediately preceding frame may be used as an 
estimated polygon without performing approximation at 
step S81. In addition, if an approximation polygon is 

25 first obtained for only one frame or a small number of 

frames, an approximation in the immediately preceding 
frame may be used as an estimated polygon without 
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performing approximation at step S81. 

Next, at step S83, an object region for a frame to 
be processed is read, and each vertex of the estimated 
approximation polygon is moved in accordance with such 
5 object region. An example of the object region for a 

inputted frame to be processed is designated by 
reference numeral 603 shown in FIG. 13. 

There are a variety of moving methods including 
moving an object to a position on an object contour 
10 that is the closest from such estimated position for 

each vertex or moving an object by using a parameter 
optimizing method such as gradient approach or Newton 
approach so as to minimize an approximation error. 

If a plurality of objects (i.e., approximation 
15 polygons) exist in the adjacent frames, as in the first 

embodiment, an object region determined to be 
associated with an object in the targeted frame is used, 

An example wherein vertexes are thus moved to be 
associated with an object is designated by reference 
20 numeral 604 shown in FIG. 13. In addition, an example 

of the moved and generated approximation polygon is 
designated by reference numeral 605 shown in FIG. 13. 

The present embodiment is similar to the first 
embodiment in function approximation processing using 
25 the function approximation unit 103. 

In addition, it is possible to record the 
coordinate value of each vertex of the approximation 



polygon in each frame as it is in the object region 
data without performing function approximation using 
the function approximation unit 103. In the latter 
case, the function approximation unit 103 may not be 
provided . 

As has been described above, according to the 
present embodiment, there is provided an object region 
data generating method and apparatus for generating 
object region data that describes data concerning a 
region of an arbitrary object in moving picture data 
over a plurality of continuous frames, wherein, based 
on data indicating a trajectory of position data on a 
plurality of vertexes of an approximation polygon that 
approximates the region for the object from the already 
generated starting frame over the immediately preceding 
frame, position data on a plurality of vertexes of the 
approximation polygon that approximates the region for 
the object in the frame to be processed is estimated, 
and is modified so as to be associated with a region 
for the object in the frame to be processed, whereby 
processing is performed for generating data indicating 
a trajectory of position data on a plurality of 
vertexes of an approximation polygon that approximates 
a region for the object from the starting frame to the 
target frame, and the object region data is generated 
based on a trajectory of the plurality of vertexes 
obtained by repeatedly performing such processing over 





m 
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the plurality of frames. 

In order to estimate position data on vertexes of 
an approximation polygon in a frame to be processed, 
the trajectory of position data on vertexes of the 
5 approximation polygon over a plurality of already 

generated frames is approximated by function so as to 
estimate the position data on vertexes of the 
approximation polygon in the already generated closest 
frame. 

10 In order to estimate position data on vertexes of 

the approximation polygon, the position data on the 
vertex of the approximation polygon in the already 
generated closest frame may be used. 

In order to estimate position data on vertexes of 

15 the approximation polygon, the position of the center 

of gravity of the frame may be estimated from the 
center of gravity of the approximation polygon over a 
plurality of the already generated frames so as to use 
the position data on the vertexes of the approximation 

20 polygon when the approximation polygon in the already 

generated closest frame is moved to the positional 
center of gravity. 

As described above, according to the present 
invention, it is possible to approximate the size, 

25 shape, movement, or deformation and the like, of a 

region of a desired object in a moving picture, as a 
polygon, with high precision and high speed. In 
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addition, the region for the desired object in the 
moving picture is described in a small amount of data, 
making it possible to facilitate object generation or 
data processing. As a result, in a moving picture 
5 database or a supervisory system and the like, the 

approximated polygon is used for object recognition, 
classification, search or the like. 

Further, according to the present invention, 
preferable function approximation is performed even if 
10 there is a possibility that the number of vertexes 

^ differs depending on each frame or there is a high 

SB possibility that vertexes between frames are associated 

5 with each other by object deformation and movement or 

i - 

isj the like. 

jfi 15 The present invention according to the apparatus 

1^ is established as that according to the method, and 

vice versa. 

Furthermore, the present invention can be 
implemented as a computer readable recording medium 
20 that records a program for causing a computer to 

executing predetermined program codes (or causing a 
computer to function as predetermined means or causing 
a computer to implement a predetermined function) . 
As has been described above, according to the 
25 present invention, it is possible to approximate a 

region of a desired object in a moving picture as a 
polygon with high precision and high speed and the 



region of the desired object in the moving picture is 
described in small amount of data, making it possible 
to facilitate object generation or data processing. 

Additional advantages and modifications will 
readily occur to those skilled in the art. Therefore, 
the present invention in its broader aspects is not 
limited to the specific details, representative devices, 
and illustrated examples shown and described herein. 
Accordingly, various modifications may be made without 
departing from the spirit or scope of the general 
inventive concept as defined by the appended claims and 
their equivalents . 



